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Abstract  -  The  Air  Force  future  vision  for  information 
fusion  is  presented  in  the  context  of  results  of  a  study  of 
long-term  challenges  in  the  science  and  technology  of 
command  and  control  and  the  programs  of  the  Air  Force 
designed  to  meet  them.  A  new  approach  to  research  is 
presented  that  takes  advantage  of  scientific  challenge 
problems  that  are  operationally  validated  in 
collaboration  with  research  laboratories  in  government 
and  industry. 
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1  Introduction 

The  Air  Force  Office  of  Scientific  Research  (AFOSR)  is  the 
single  manager  of  basic  research  for  the  U.S.  Air  Force 
(USAF).  Basic  research  is  defined  in  the  U.S.  Defense 
Department  as  the  most  fundamental  of  work  along  a 
research  and  development  spectrum  from  knowledge 
discovery  to  technology  demonstration.  The  AFOSR 
supports  many  areas  of  science,  but  focuses  its  support 
on  topics  expected  strongly  to  contribute  to  the 
technology  needs  of  the  USAF.  These  programs  are 
executed  largely  through  grants  and  contracts  to  industrial 
and  academic  researchers. 

The  AFOSR  program  in  Information  Fusion,  the  topic 
of  this  special  session,  addresses  fundamental  issues  in 
the  ways  that  information  can  be  best  combined  and  used 
to  support  decision-making  and  evaluation  of  decision 
outcome.  Other  research  funding  organizations  address 
these  and  similar  issues,  but  the  AFOSR  program  is  unique 
for  attempting  to  use  operationally  valid  challenge 
problems  to  focus  research  and  to  quantify  the  benefits  of 
technologies  fostered  by  the  research.  The  research 
community  acting  alone  cannot  accomplish  this  task.  The 
AFOSR  program  involves  the  cooperation  of  technology 
developers  and  operational  users  in  government  and 
industry . 

Other  AFOSR  programs  relate  to  Information  Fusion, 
but  do  not  explicitly  address  issues  of  level  2  and  higher 


fusion  (See  Figure  1),  so  are  not  described  here  in 
significant  detail.  These  related  AFOSR  programs  address 
issues  of  multi-  and  hyper-spectral  sensing,  signal 
processing,  automatic  target  recognition,  information 
assurance,  guidance  and  control,  artificial  intelligence,  and 
team  command  and  control.  A  brief  description  of  these 
and  other  programs  may  be  found  in  the  AFOSR  Broad 
Agency  Announcement  available  at  www.afosr.af.mil. 


0  —  Sub- Object  Data  Association  & 
Estimation 

1  —  Object  Refinement 

2  —  Situation  Refinement 

3  —  Impact  Assessment 

4  —  Process  Refinement 


Figure  1.  Levels  of  Information  Fusion. 

2  Air  Force  Future  Vision 

The  AFOSR  employs  a  number  of  mechanisms  to  help 
define  its  current  and  future  programs.  The  AFOSR 
program  in  Information  Fusion,  for  example,  has  relied  on 
studies  conducted  by  the  Air  Force  Scientific  Advisory 
Board  (SAB)  to  help  shape  programs  and  relate  the 
scientific  product  to  future  technology  needs  of  the  Air 
Force.  This  relationship  between  research  on  Information 
Fusion  and  the  future  technology  needs  is  described  in  a 
number  of  SAB  studies  outlined  here.  (Some  of  these 
studies  are  available  at  http://www.sab.hq.af.mil.) 

A  seminal  study  of  the  SAB,  “Report  on  Building  the 
Joint  Battlespace  Infosphere”  (JBI)  [1],  defines  a  future 
combat  information  management  system  that  creates  and 
maintains  a  common  operating  picture  for  decision  support 
at  multiple  echelons.  As  depicted  in  Figure  2  and 
described  in  the  report,  the  JBI  provides  a  substrate  for 
integrating  existing  and  planned  information  systems  for 
command  and  control,  and  for  using  the  integrated  result 
to  do  a  lot  more  for  intelligence  preparation,  operational 
planning,  and  mission  assessment. 
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The  JBI  report  outlines  a  number  of  scientific  and 
technical  challenges  that  include  some  elements  of 
communication  and  knowledge  engineering,  but  the  report 
largely  focuses  on  issues  of  higher  levels  of  Information 
Fusion.  Specifically,  the  JBI  report  calls  for  technologies 
capable  of  transforming  information  to  operational 
knowledge.  In  such  contexts  as  intelligence  preparation  of 
the  battlefield,  for  example,  creation  of  a  common  operating 
picture  is  seen  to  rely  on  the  aggregation  of  surveillance 
from  diverse  collection  platforms  and  its  merger  with 
existing  information  on  the  disposition  of  forces,  weather, 
and  terrain  features.  More  significantly,  the  JBI  report 
outlines  ways  that  aggregated  information  might  be 
customized  to  user  needs  (as  represented  in  models  of 
user  goals,  plans,  and  preferences),  to  support  planning 
and  execution. 


Figure  2.  Depiction  of  Joint  Battlespace  Infosphere 

The  emphasis  on  user  models  in  the  JBI  report 
represents  a  severe  scientific  and  technological  challenge 
and  helps  define  the  AFOSR  program  in  Information 
Fusion.  When  available,  user  models  largely  determine 
‘what’  to  compute  (as  opposed  to  ‘how’  to  compute,  or 
communicate,  or  store  knowledge  -  issues  addressed  in 
other  related  research  programs)  and  completely  determine 
the  utility  of  the  common  operating  picture.  User  models 
expressed  as  a  number  of  decision  rules  has  been 
estimated  in  certain  operational  contexts  and  is  quite  large, 
on  the  order  of  ten  million  rules.  Using  technologies 
currently  available,  approximately  one  million  rules  per 
year  can  be  ascertained.  This  gap  means  that  a  JBI  will 
forever  remain  beyond  reach  of  current  technologies 
because  users  alter  their  models  in  response  to  a 
constantly  changing  operational  environment.  The 
AFOSR  program  incorporates  scientific  study  of  methods 
to  extract  from  human  experts  those  decision  rules  that 
help  determine  what  information  ought  to  be  fused. 


Subsequent  studies  of  the  SAB  have  refined  the  JBI 
notion  along  several  important  lines.  One  SAB  study 
conducted  in  2001  and  planned  for  release  in  May  2002, 
titled  “Migration  of  Data  Bases  for  Command  and  Control” 

[2] ,  was  established  to  addresses  the  need  for  JBI  access 
to  all  data  currently  used  in  DoD  systems  of  command  and 
control.  According  to  the  terms  of  reference  for  this 
study,  successful  implementation  of  JBI  will  require  the 
continued  availability  of  legacy  information  systems  that 
presumably  would  require  their  migration  to  a  common  JBI 
environment.  The  final  report  from  this  study  may 
illuminate  technical  challenges  in  knowledge  engineering 
but,  because  information  technology  data  standards  do 
not  yet  exist  (and  may  never  exist  for  uncontrolled  data 
sources),  the  goal  of  AFOSR’ s  Information  Fusion 
program  involves  the  creation  of  actionable  information 
from  data  in  a  large  variety  of  structures  and  formats. 

A  second  recent  study  of  the  SAB  explicitly 
addresses  sensor  and  data  information  fusion  to  meet 
future  operational  concepts  for  difficult  targets,  such  as 
targets  that  are  moving  or  concealed  or  buried.  A  report 
from  this  study,  “Sensor  Technology  for  Difficult  Targets” 

[3] ,  is  also  scheduled  for  release  in  May  2002.  It’s 
anticipated  that  target  types  addressed  in  this  study  will 
require  a  level  of  knowledge  and  decision-making  that 
goes  beyond  simple  identification  and  recognition  (i.e.  that 
requires  information  fusion  at  levels  2  and  3),  because 
target  defeat  in  some  cases  requires  exquisite  modeling  of 
the  functional  area  or  the  functional  site  where  the  target  is 
found.  To  address  these  issues  of  target  description,  the 
AFOSR  program  in  Information  Fusion  involves  the 
creation  and  maintenance  of  functional  area  models  that 
support  pre-planning,  operational  decision-making,  plan 
execution,  and  outcome  assessment. 

Lastly,  an  ongoing  study  of  the  SAB,  “Predictive 
Battlespace  Awareness  to  Improve  Military  Effectiveness” 

[4] ,  addresses  processes  and  approaches  for  course  of 
action  planning  that  involve  active  management  of 
collection  resources  up  to  and  including  real  time  sensor 
management.  The  ongoing  study  considers  intelligence, 
surveillance,  and  reconnaissance  systems  whose 
improved  management  could  yield  improved  quality  of 
decisions.  As  a  result,  the  emphasis  of  this  study  appears 
to  involve  coordination  of  collection  resources  already 
available  rather  than  the  definition  of  new  information 
sources.  To  address  anticipated  future  needs  for  real-time 
management  of  coordinated  systems  of  information 
gathering,  supervised  decision-making  and  execution,  and 
assessment,  the  AFOSR  program  in  Information  Fusion 
addresses  cooperative  planning  in  supervised 
autonomous  systems. 
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3  Scientific  Challenge  Problems 

As  a  whole,  the  studies  of  the  Air  Force  Scientific 
Advisory  Board  listed  above  define  a  future  vision  for 
higher  levels  of  information  fusion.  This  vision  requires 
new  technologies  to  accommodate  increasing  volumes  of 
information,  increasing  real-time  management  of 
information  assets,  and  increasing  sensitivity  to  the 
information  demands  of  users  at  all  echelons.  The  AFOSR 
program  in  Information  Fusion  directly  addresses  these 
issues  using  an  approach  based  on  challenge  problems 
described  in  following  sections. 

The  three  challenge  problems  derive  from  an  AFOSR- 
sponsored  workshop,  held  in  October  2000,  that  surveyed 
the  state  of  technology  in  the  context  of  user  needs  and 
research  capabilities  of  government,  industry,  and 
academia.  At  this  workshop,  discussions  revealed  three 
pacing  technology  issues,  for  which  significant 
enhancements  in  operational  capability  could  be  achieved 
by  advancing  identified  technologies  using  one  or  another 
approach  employed  in  academic  research.  These  pacing 
issues  are  captured  in  the  challenge  problems.  The 
workshop  also  revealed  that  operational  systems  for 
Information  Fusion  at  higher  levels  either  did  not  exist  or 
were  in  a  state  of  initial  development.  The  lack  of  fielded 
systems  suggested  that  a  collaborative  approach  among 
stake -holding  communities  (of  researchers,  technologists, 
and  users)  might  be  most  effective,  because  each 
community  was  oriented  toward  a  future  system  that  did 
not  exist  and  whose  existence  would  depend  on 
technologies  in  need  of  some  maturation  and  evolution 
through  research. 


What  You  Do  What  You  Get 


Operators 


Developers 


Researchers 


Figure  3.  Challenge  Problem  Win -Win -Win 

AFOSR  devised  a  challenge  problem  approach  to 
foster  collaboration  between  researchers,  technologists, 
and  users.  To  provide  incentive  for  collaboration,  the 
approach  provides  each  of  the  constituent  communities 
with  something  of  value  (See  Figure  3).  The  user 
community  provides  benchmark  measures  of 
effectiveness,  cooperates  in  technology  validation,  and 
receives  field -able  new  technologies.  The  technology 


•  Support  System  Analysis 
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•  Rapid  Technology  Insertion 

•  Build  Simulator 
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•  Common  Tools 
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community  establishes  research  challenge  problems  and 
receives  a  new  tool  for  operationally  valid  technology 
assessment  (this  permits  government  laboratories  to 
certify  commercial  designs).  The  research  community 
addresses  the  challenge  problems  and  receives  a  means  of 
determining  the  relative  significance  of  new  discoveries. 
Presently,  comparing  research  results  is  difficult  because 
the  dimensionality  and  the  metric  for  improvements  are 
absent.  To  partially  alleviate  this  problem,  properly 
constructed  challenge  problems  contain  operationally 
valid  performance  metrics. 

The  three  challenge  problems  under  development  in 
the  AFOSR  program  each  address  one  of  the  pacing 
issues  in  higher  levels  of  Information  Fusion  defined  at  the 
workshop  described  above.  At  a  very  general  level,  the 
pacing  issues  concern  space  (the  need  to  automate  mining 
of  tremendous  fused  volumes  of  image  and  other  data), 
time  (the  need  for  rapid  dynamic  re -planning  among  a 
constellation  of  supervised  autonomous  platforms),  and 
intent  (the  need  to  adjust  a  common  operating  picture  to 
meet  changing  demands  of  decision-making  teams).  Cast 
in  this  general  way,  the  challenge  problems  also  map  to  the 
reports  of  the  Air  Force  Scientific  Advisory  Board  outlined 
above  -  respectively,  to  the  reports  on  Sensors  for 
Difficult  Targets,  Predictive  Battle  space  Awareness,  and 
Joint  Battlespace  Infosphere. 

3.1  Spatial:  Image  Analysis 

This  challenge  problem  addresses  the  need  to  merge 
multiple  hyper-spectral  views  of  an  area  with  other 
information  about  the  objects  in  that  region.  The  scientific 
issues  include  creation  and  maintenance  of  a  common 
operating  picture  that  includes  high-level  descriptions  for 
functional  elements  under  view.  Technologies  are  needed 
for  robustly  adapting  to  scene  changes  and  flexibly 
responding  to  user  information  needs.  Work  on  this 
challenge  can  take  advantage  of  other  research  on  data 
storage,  parallel  computation,  high  bandwidth 
communications,  automatic  pattern  recognition, 
inferencing,  and  human  interface.  The  effectiveness  of 
theory  and  modeling  in  this  area  can  be  assessed  using 
measures  that  evaluate  the  computational  resources 
required  (a  cost)  to  provide  accurate  and  precise 
functional  area  descriptions  (a  benefit)  using  all  sources  of 
available  information  (a  presumed  benefit).  The  primary 
beneficiary  of  associated  technology  advances  is  the 
image  analyst.  Allen  Waxman  describes  the  current  status 
of  work  on  this  challenge  problem  in  the  following  talk  [5]. 

3.2  Temporal:  Cooperative  Assessment 

This  challenge  problem  addresses  the  need  for  efficient 
goal-seeking  behavior  in  a  group  of  cooperating  mobile 
platforms.  The  scientific  issues  include  dynamic 
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replanning  in  uncertain  environments  and  distributed 
knowledge  management.  Technologies  are  needed  to 
maintain  multiple  hierarchical  control  loops  and  to  optimize 
platform  mo  vements  quickly  in  response  to  environmental 
discoveries  and  changing  supervisory  goals.  Work  on 
this  challenge  can  take  advantage  of  other  research  on 
sensors,  signal  processing,  automatic  target  recognition, 
robotics,  cooperative  control,  information  assurance,  and 
adaptive  network  architectures.  Advances  in  theory  and 
modeling  in  this  area  can  be  measured  in  terms  of  costs  in 
communication  bandwidth  and  length  of  travel  required  for 
limited  mobile  resources  to  build  and  maintain  an  accurate 
operating  picture  of  the  environment  surveyed.  Primary 
users  of  associated  technologies  do  not  exist  today,  but 
system  developers  are  actively  seeking  design  solutions 
for  managed  networks  of  mobile  platforms.  Current  status 
of  this  challenge  problem  is  described  in  two  later  talks, 
one  by  Katia  Sycara  [6],  the  other  by  Kevin  O’Neal  and 
Robert  Murphey  [7]. 

3.3  Intentional:  Crisis  Management 

This  challenge  problem  addresses  the  need  to  provide 
human  experts  with  decision  support  and  to  partially 
automate  command  and  control  decision-making.  The 
scientific  issues  concern  the  construction  of  decision  rules 
that  reflect  expert  intent  and  mimic  those  of  human 
members  in  a  command  and  control  team.  Technologies 
are  needed  to  create  decision  rules  very  rapidly  under 
expert  human  supervision  and  to  implement  the  decision 
rules  as  a  human  replacement  or  augmentation  in  context 
of  team  decision-making.  Work  on  this  challenge  can  take 
advantage  of  related  work  in  other  scientific  areas 
including:  team  command  and  control,  cognitive  science, 
artificial  intelligence,  knowledge  engineering,  operations 
research,  and  modeling  and  simulation.  Advances  in 
theory  and  modeling  in  this  area  can  be  assessed  in  terms 
of  the  number  of  human  team  members  displaced  while 
maintaining  fixed  level  of  control  over  a  dynamic 
environment.  The  primary  customer  of  associated 
technologies  is  the  manager  of  an  operational  command 
and  control  center  or  a  crisis  management  center.  One 
following  presentation  by  James  Llinas  [8]  includes  a 
strategy  for  assessing  robustness  of  Information  Fusion 
techniques  in  the  context  of  disaster  decision  support,  and 
a  later  presentation  by  Daniele  Nardi  and  others  [9] 
describes  work  on  the  RoboCup  Rescue  challenge  problem 
for  disaster  monitoring. 

4  Programmatics 

The  AFOSR  program  in  Higher  Levels  of  Information 
Fusion  supports  collaborations  between  academia, 
industry,  government  laboratories,  and  the  user 
community.  This  program  is  woven  into  other  AFOSR 
programs  addressing  many  other  scientific  issues  related 


to  Information  Fusion.  Plans  for  support  of  additional 
research  in  this  area  include  opportunities  provided  by 
programs  that  support  research  centers  and  collaborations 
between  academia  and  industry. 

4.1  Partnerships 

The  AFOSR  establishes  “Partnerships  for  Research 
Excellence  and  Transition”  (PRET)  to  support  research 
centers  involving  collaborations  between  academia  and 
industry.  The  PRET  centers  are  established  in  scientific 
areas  where  the  pace  of  knowledge  discovery  is  high  and 
sufficiently  mature  to  provide  strong  opportunities  for 
technology  development.  Such  a  state  characterizes 
scientific  work  on  Information  Fusion  because  new 
theories  and  models  are  often  expressed  in  software  that 
can  be  quickly  inserted  as  modules  into  existing  or  newly 
designed  systems  for  enhanced  capability.  Recognizing 
that  the  most  promising  approaches  to  Information  Fusion 
have  yet  to  be  identified  and  that  sorting  among  the 
perhaps  conflicting  claims  of  various  approaches  might  be 
difficult,  the  PRETs  for  Information  Fusion  address 
challenge  problems  as  described  above.  The  challenge 
problems  provide  a  mechanism  for  focusing  research  and 
for  determining  when  scientific  advances  truly 
revolutionize  current  capabilities.  To  implement  challenge 
problems  that  include  operationally  valid  performance 
criteria,  the  AFOSR  has  enhanced  the  typical  PRET  for 
Information  Fusion  to  include  collaborators  from  the  user 
community  and  certain  Technology  Directorates  of  the  Air 
Force  Research  Laboratory.  The  remainder  of  this  special 
session  includes  presentations  by  scientific  and 
technological  collaborators  in  PRET  activity.  As 
described  above,  these  presentations  describe  the 
collaborative  activity  of  partners  in  academia,  industry, 
and  government  representing  researchers,  technologists, 
and  users. 

4.2  Woven  Disciplines 

Many  scientific  disciplines  contribute  to  advanced 
capability  for  Information  Fusion  or  are  required  to 
implement  functional  systems.  In  other  programs,  the 
AFOSR  supports  research  in  many  disciplines  related  to 
Information  Fusion  and  maintains  relationships  with  those 
programs  through  joint  workshops,  visits,  and  exchanges 
of  personnel.  The  few  examples  that  follow  illustrate  that 
the  science  of  Information  Fusion  can  take  advantage  of 
advances  in  other  areas  and  that  implementations  of 
Fusion  technologies  may  require  advances  in  certain  other 
disciplines.  Work  on  team  performance,  for  example,  can 
provide  the  Fusion  community  with  specification  of  an 
operating  picture  and  with  description  of  how  team 
members  best  operate  for  fast  accurate  decision-making. 
Work  on  Sensors,  Automatic  Target  Recognition,  and 
Tracking  can  provide  more  robust  inputs  to  the  fusion 
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process  and  offload  requirements  for  sophisticated 
inferencing  that  would  otherwise  involve  fused 
information.  Work  on  guidance  and  control  of 

autonomous  vehicles  can  provide  standards  for  interactive 
control  of  these  vehicles  by  fusion-based  management 
systems.  Work  on  communications,  including  information 
assurance,  can  provide  bandwidth  that  might  be  required 
to  construct  and  deliver  a  common  operating  picture. 
Lastly,  work  on  large  databases  and  knowledge 
engineering  can  provide  needed  repositories  for  the  grist 
of  fusion  mills.  To  better  link  advances  in  these 
disciplines  with  those  in  Information  Fusion,  the 
maturation  and  delivery  of  the  challenge  problems  remains 
a  high  priority  in  this  program. 

4.3  Program  Plans 

The  AFOSR  plans  to  increase  support  for  research  on 
higher  levels  of  information  fusion  through  funding  of 
additional  research  centers,  smaller  collaborations  between 
academia  and  industry,  and  international  research. 
Opportunity  to  propose  additional  centers  may  be  offered 
through  the  Multidisciplinary  Research  Program  of  the 
University  Research  Initiative  (MURI)  typically 
announced  during  the  summer  of  each  year.  The  MURI 
centers  involve  academic  institutions  only,  funded  up  to 
one  million  per  year  for  up  to  five  years.  Opportunities  for 
smaller  collaborations  between  academia  and  industry  may 
be  provided  through  the  Small  Business  Technology 
Transfer  (STTR)  program  typically  announced  at  the 
beginning  of  each  calendar  year.  This  program  provides 
up  to  $600K  in  early  stage  research  and  development 
funding  directly  to  small  companies  working  cooperatively 
with  researchers  at  universities  and  other  research 
institutions.  International  research  is  supported  under  the 
AFOSR  “International  Research  Initiatives”  program.  This 
program  is  announced  annually,  usually  in  the  fall,  and 
supports  efforts  such  as  that  presented  later  in  this 
session  by  collaborators  at  the  University  of  Rome,  Italy. 

As  the  challenge  problems  mature,  the  AFOSR  plans 
to  increase  coordination  with  other  Federal  funding 
agencies  that  provide  funding  for  research  and  technology 
development  in  higher  levels  of  information  fusion. 

5  Summary 

The  AFOSR  program  in  Higher  Levels  of  Information 
Fusion  supports  research  directed  at  future  Air  Force 
needs  conveyed  in  recent  reports  of  the  Air  Force 
Scientific  Advisory  Board.  The  program  implements 
research  challenge  problems  to  focus  multidisciplinary 
research  in  this  area  and  to  establish  metrics  for  comparing 
the  magnitude  of  new  findings.  Implementation  of 
challenge  problems  with  operational  validity  requires 
research  collaboration  with  government  laboratories  and 


the  user  community.  AFOSR  supports  hese  needed 
collaborations  through  “Partnerships  for  Research 
Excellence  and  Transition”  combined  with  other  funding 
mechanisms,  and  plans  to  expand  its  support  for  research 
on  higher  levels  of  information  fusion. 

The  current  scope  of  AFOSR’ s  program  in  Higher 
Levels  of  Information  Fusion  is  illuminated  by  the  several 
papers  that  follow  in  this  special  session. 
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